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Abstract 

In this paper, we complete the global qualitative analysis of a quartic family of 
planar vector fields corresponding to a rational Holling-type dynamical system which 
models the dynamics of the populations of predators and their prey in a given 
ecological or biomedical system. In particular, studying global bifurcations, we prove 
that such a system can have at most two limit cycles surrounding one singular point. 
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1 Introduction 


In this paper, we consider a quartic family of planar vector fields corresponding 
to a rational Holling-type dynamical system which models the dynamics of 
the populations of predators and their prey in a given ecological or biomedical 
system and which is a variation on the classical Lotka-Volterra system. For 
the latter system the change of the prey density per unit of time per predator 
called the response function is proportional to the prey density. This means 
that there is no saturation of the predator when the amount of available prey 
is large. However, it is more realistic to consider a nonlinear and bounded 
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response function, and in fact different response functions have been used in 
the literature to model the predator response; see u-B, ra-ia. ra. 

For instance, in [20], the following predator-prey model has been studied: 


X = x{a — Xx) — yp{x) (prey), 
y = —6y + yq{x) (predator). 


( 1 . 1 ) 


The variables x > 0 and y > 0 denote the density of the prey and preda¬ 
tor populations respectively, while p{x) is a non-monotonic response function 
given by 


p{x) 


mx 

ax^ -1- /3x -I- 1 ’ 


( 1 . 2 ) 


where a, m are positive and where (3 > —2y/a. Observe that in the absence 
of predators, the number of prey increases according to a logistic growth law. 
The coefficient a represents the intrinsic growth rate of the prey, while A > 0 
is the rate of competition or resource limitation of prey. The natural death 
rate of the predator is given by 5 > 0. In Cause’s model the function g(x) is 
given by q{x) = cp{x), where c > 0 is the rate of conversion between prey and 
predator EDI- 


In 0 , 11 ], the following family has been investigated: 


^ = -). 

\ ax^ + /3x + l J 

y = -y{^ + py -y—^ , 

\ ax^ + px + IJ 


(1.3) 


where a > 0, (3 > 5 > 0, A > 0, and /x > 0 are parameters. Note that 

(1.3) is obtained from (1.1) by adding the term —py’^ to the second equation 
and after scaling x and y, as well as the parameters and the time t. In this way, 
it has been taken into account competition between predators for resources 
other than prey. The non-negative coefficient p is the rate of competition 
amongst predators. Systems (1.1)-(1.3) represent predator-prey models with 
a generalized Holling response functions of type IV. 


In [T7], it has been considered the following generalized Cause predator-prey 
system 


X = rx(l — x/k) — yp{x), 
y = yi-d + cp{x)) 


(1.4) 


2 





with a generalized Holling response function of type III: 


p{x) 


mx^ 

ax‘^ + bx + 1 


(1.5) 


This system, where x > 0 and y > 0, has seven parameters: the parameters 
a, c, d, k, m, r are positive and the parameter b can be negative or non¬ 
negative. The parameters a, b, and m htting parameters of response function. 
The parameter d is the death rate of the predator while c is the efficiency of 
the predator to convert prey into predators. The prey follows a logistic growth 
with a rate r in the absence of predator. The environment has a prey capacity 
determined by k. 


The case 6 > 0 has been studied earlier; see the references in HZI. The case 
6 < 0 is more interesting: it provides a model for a functional response with 
limited group defence. In opposition to the generalized Rolling function of 
type IV studied in 0. II. izni, where the response function tends to zero as 
the prey population tends to inhnity, the generalized function of type III tends 
to a non-zero value as the prey population tends to inhnity. The functional 
response of type III with 6 < 0 has a maximum at some point; see na. When 
studying the case 6 < 0, one can hnd also a Bogdanov-Takens bifurcation of 
codimension 3 which is an organizing center for the bifurcation diagram of 
system (1.10)-(1.5) [T7] . 


After scaling x and y, as well as the parameters and the time t, this system 
can be reduced to a system with only four parameters (a, (3, 6, p) [T7j : 


where 


X = px{l — x) — yp{x), 
y = y{-6 + p{x)), 


p{x) 


ax^ + (5x + 1 


( 1 . 6 ) 


(1.7) 


Note that many studies of discrete-time predator-prey models have been done; 
see, e. g., [TS] and the references therein. In particular, the following discrete 
predator-prey system with a generalized Rolling response function of type III 
has been investigated in [18] : 


Xfc+i = Xfc + Oxk f p(l - Xfc)(xfc - A)-—TT 

\ axi + pxk + 1 

Vk+i = yk + dyk ( —S H-2 o —TT ) ’ 

\ axi P- pxk + 1 / 

where 0 > 0 is the step size in the forward Euler scheme. 


( 1 . 8 ) 


3 






In this paper, we study the system 


\ ax^ + (3x + l) 

y = -yU + ^^y -jtV— r) 

\ ax^ + px + 1) 


(prey), 


(predator). 


(1.9) 


where x > 0 and ?/ > 0; a > 0, —cx) < (3 < +oo, 5 > 0, A > 0, and /i > 0 are 
parameters. 


Rational system (1.9) can be written in the form of a quartic dynamical system 


X = x((l — Ax)(ax^ + /3x + 1) — xy) = P, 
ij = -y{{5 + ij,y){ax^ + l3x + l) - x^) = Q. 


( 1 . 10 ) 


Together with (1.10), we will also consider an auxiliary system (see PQ, [T9] i 

x = P-pQ, 2/= <5 + 7^1 (1-11) 

applying to these systems new bifurcation methods and geometric approaches 
developed in [1]-[T5] and completing the qualitative analysis of (1.9). 


2 Basic facts on singular points and limit cycles 


The study of singular points of system (1.9) will use two index theorems by 
H. Poincare; see [1]. But hrst let us dehne the singular point and its Poincare 
index [T]. 


Definition 2.1. A singular point of the dynamical system 

x = P{x,y), y = Q{x,y), (2.1) 

where P{x, y) and Q{x, y) are continuous functions (for example, polynomials), 
is a point at which the right-hand sides of (2.1) simultaneously vanish. 


Definition 2.2. Let S' be a simple closed curve in the phase plane not passing 
through a singular point of system (2.1) and M be some point on S. If the 
point M goes around the curve S in positive direction (counterclockwise) 
one time, then the vector coinciding with the direction of a tangent to the 
trajectory passing through the point M is rotated through the angle 27rj 
(j = 0, ±1, ±2,...). The integer j is called the Poincare index of the closed 
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curve S relative to the vector field of system (2.1) and has the expression 


1 rPdQ-QdP 

^ ~^J P2 + Q2 • 

S 

According to this definition, the index of a node or a focus, or a center is equal 
to +1 and the index of a saddle is —1. 


Theorem 2.1 (First Poincare Index Theorem). If N, Nf, Nc, and C are 

respectively the number of nodes, foci, centers, and saddles in a finite part of 
the phase plane and N' and C are the number of nodes and saddles at infinity, 
then it is valid the formula 

N + NfPN^ + N' = C + C' + 1. 


Theorem 2.2 (Second Poincare Index Theorem). If all singular points 
are simple, then along an isocline without multiple points lying in a Poincare 
hemisphere which is obtained by a stereographic projection of the phase plane, 
the singular points are distributed so that a saddle is followed by a node or a 
focus, or a center and vice versa. If two points are separated by the eguator 
of the Poincare sphere, then a saddle will be followed by a saddle again and a 
node or a focus, or a center will be followed by a node or a focus, or a center. 


Consider polynomial system (2.1) in the vector form 

x = f{x,n), (2.2) 

where x G R^; pL G R"; / G R^ (/ is a polynomial vector function). 

Let us recall some basic facts concerning limit cycles of (2.2). But first of all, 
let us state two fundamental theorems from theory of analytic functions [5]. 


Theorem 2.3 (Weierstrass Preparation Theorem). Let F{w,z) be an 
analytic in the neighborhood of the point (0, 0) function satisfying the following 
conditions 


F(0,0) = 0, 


dF(0,0) 

dw 


d^-iF(0,0) 

’ d^~^w 


0 ; 


d^F(0,0) 

d^w 


^0. 


Then in some neighborhood |tc| < e, \z\ < S of the points (0,0) the function 
F{w,z) can be represented as 

F{w, z) = {w^ + Ai{z)w^~^ + ... + Ak-i{z)w + Ak{z))^{w, z), 
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where z) is an analytic function not equal to zero in the chosen neigh¬ 
borhood and Ai{z),, Ak{z) are analytic functions for \z\ < 6. 

From this theorem it follows that the equation F{w,z) = 0 in a. sufficiently 
small neighborhood of the point (0, 0) is equivalent to the equation 

+ Ai[z)w^ ^ + ... + Ak—i[z)w + Ak[z) = 0 , 

which left-hand side is a polynomial with respect to w. Thus, the Weierstrass 
preparation theorem reduces the local study of the general case of implicit 
function w{z), dehned by the equation F{w,z) = 0, to the case of implicit 
function, dehned by the algebraic equation with respect to w. 


Theorem 2.4 (Implicit Function Theorem). Let F{w,z) be an analytic 
function in the neighborhood of the point ( 0 , 0 ) and F( 0 , 0 ) = 0 , F^{0, 0 ) 7 ^ 0 . 

Then there exist 5 > 0 and e > 0 such that for any z satisfying the condition 
\z\ < 6 the equation F{w, z) = 0 has the only solution w = f{z) satisfying the 
condition \f{z)\ < e. The function f{z) is expanded into the series on positive 
integer powers of z which converges for \z\ < 6, i. e., it is a single-valued 
analytic function of z which vanishes at z = 0. 

Assume that system (2.2) has a limit cycle 


Lo:x = (fQ^t) 


of minimal period Tq at some parameter value E R" (Fig. 1). 




FIG. 1. The Poincare return map in the neighborhood of a multiple limit cycle. 

Let I be the straight line normal to Lq at the point = ¥^ 0 ( 0 ) s be 
the coordinate along I with s positive exterior of Lq. It then follows from the 
implicit function theorem that there is a 5 > 0 such that the Poincare map 
h{s,p) is dehned and analytic for |s| < 6 and ||^ — /XqII < Besides, the 
displacement function for system (2.2) along the normal line I to Lq is dehned 
as the function 

d{s, p) = h(s, p) — s. 
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In terms of the displacement fnnction, a mnltiple limit cycle can be defined as 
follows [5]. 

Definition 2.3. A limit cycle Lq of (2.2) is a multiple limit cycle iff d{0, /Xq) = 
dr{0,i-iQ) = 0 and it is a simple limit cycle (or hyperbolic limit cycle) if it is 
not a mnltiple limit cycle; furthermore, Lq is a limit cycle of multiplicity m iff 

c^(0, Ato) = <^r(0, pto) = • • • = Ato) = 4™nO,Aio) 7^ 0- 


Note that the multiplicity of Lq is independent of the point pg ^ -^o through 
which we take the normal line 1. 

Let us write down also the following formulas which have already become 
classical ones and determine the derivatives of the displacement function in 
terms of integrals of the vector held / along the periodic orbit ^f^it) |5]: 

To 

ds{0,tio) = exp^V ■/((Po(^), Mo) dt - 1 (3.2) 

0 


and 

4,(0 >^o) = 

||/(y~(o4o)ll 4 (-4 A/„,(¥>„((). Mo) dt ( 3 , 3 ) 

for j = where Uq = ±1 according to whether Lq is positively or 

negatively oriented, respectively, and where the wedge product of two vectors 
X = {xi,X2) and y = (|/i, 2 / 2 ) in is dehned as 


X Ay = xiy2- X2 yi- 


Similar formulas for dss{0, and (0,Po) can be derived in terms of in¬ 
tegrals of the vector held f and its hrst and second partial derivatives along 
cpg(t). The hypotheses of theorems in the next section will be stated in terms 
of conditions on the displacement function d{s, y) and its partial derivatives 
at (0,Po) 0- 


3 Local bifurcation surfaces and the global termination principle 
for multiple limit cycles 


In this section, we restate hrst Perko’s theorems on the local existence of 
(n —m-|-l)-dimensional surfaces. Cm, of multiplicity-m limit cycles for the 
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polynomial system (2.2) with /x G R” and n > m > 2. These resnlts describe 
the topological strnctnre of the codimension (m — 1 ) bifnrcation snrfaces Cm- 
For m = 2,3,4, C 2 , C 3 , and C 4 are the familiar fold, cnsp, and swallow-tail 
bifnrcation snrfaces; for m > 5, the topological strnctnre of the snrfaces Cm is 
more complex. For instance, C 5 and Cq are the bntterfly and wigwam bifnrca¬ 
tion surfaces, respectively [19]. Since the proofs of the theorems in this section, 
describing the universal unfolding near a multiple limit cycles of ( 2 . 2 ), paral¬ 
lel the classical proofs of Catastrophe Theory, we will only state the theorems 
(see [in] for more detail). 


Definition 3.1. An (n—l)-dimensional analytic surface C 2 C R" is an (n—1)- 
dimensional fold bifurcation surface of multiplicity-two limit cycles of (2.2) 
through a point /Xg G R”, if for all £ > 0 there exists a 5 > 0 such that for each 
pL ^ C 2 with 11 ^ — ^Xgll < (5, the system ( 2 . 2 ) has a unique multiplicity-two 
limit cycle in an e-neighborhood of Lq and the system ( 2 . 2 ) undergoes a 
fold bifurcation at i. e., for ||^x — /Xq|| < S, splits into a simple stable 
and a simple unstable limit cycles in an e-neighborhood of Lq for /x on one 
side of C 2 and vanishes for pL on the other side of C 2 . Cf. Fig. 2. 



FIG. 2. The fold bifurcation surface. 

Theorem 3.1. Suppose that n > 2, that for pi = plq & RL system (2.2) has 
a multiplicity-two limit cycle Lq, and that (i^j(0,/XQ) 7 ^ 0. Then given e > 0, 
there is a 5 > Q and a unigue function g{pL 2 , • • •, /^n) with g{pif\ ..., pi^^) = 







defined and analytic for |/i 2 — <5, ..., |/i„ — /i® | < 5, such that for 

1/^2 - ..., I/in - /i® I < 

C2 ■ /il = 5 '(/i 25 • • • , /in) 

/s an (n — 1 )-dimensional, analytic fold bifurcation surface of multiplicity-two 
limit cycles of ( 2 . 2 ) through the point pL^. 

Definition 3.2. An analytic snrface C R" is an {n —2)-dimensional cusp 
bifurcation surface of multiplicity-three limit cycles of (2.2) through a point 
Hq G if for all e > 0 there exists a 5 > 0 snch that for each ^ E C 3 with 
ll/i — /igll < system (2.2) has a nnique mnltiplicity-three limit cycle in 
an e-neighborhood of Lq and the system ( 2 . 2 ) nndergoes a cnsp bifnrcation 
at i.e., is the intersection of two (n —l)-dimensional fold bifnrcation 
snrfaces of mnltiplicity-two limit cycles of ( 2 . 2 ), which intersect in a cusp 
along C 3 ; for ||/i — /Xg|| < 5 and for pL in the cuspidal region between Cf 
and Cf (shaded in Fig. 3 ), system ( 2 . 2 ) has three simple limit cycles in an 
e-neighborhood of Lq; for ||/i — /Xq|| < 5 and pi outside the cuspidal region, 
system ( 2 . 2 ) has one simple limit cycle in an e-neighborhood of Lq. Cf. Fig. 3 . 



FIG. 3. The cusp bifurcation surface. 

Theorem 3.2. Suppose that n > 3, that for pL = pL^ E system (2.2) has 
a multiplicity-three limit cycle Lq, that d^i( 0 , /Xq) 7 ^ 0 , /Xg) 7 ^ 0 
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j = 2 ,...,n, 


^ _ (9((i, dr) ^ ^ n 


Then given £ > 0, there is a S > 0 and constants Uj = ±1 forj = 2,..., n, and 
there exist unique functions hi{fj, 2 , • • •, fVn), h 2 {ti 2 , ■ ■ ■, Tn) and 5 '^(/i 2 , • • •, Tn) 
with ..., /i®) = h 2 itiP ,..., /i®) = and^g^ifiP ,..., = 

where hi and /12 ore defined and analytic for \nj — \ < 6 , j = 2, ... ,n, 

and are defined and continuous for 0 < Uj{g,j — nf^) < 6 and analytic for 
0 < ujj{nj — /i®) < 6 , j = 2, ... ,n such that 


C, : 


fil ^l(/^2) ■ ■ ■ 1 Tri) 
Tl = ^2(A''2; • • • 5 Tn) 


is an {n — 2 )-dimensional, analytic, cusp bifurcation surface of multiplicity- 
three limit cycles of ( 2 . 2 ) through the point Hq and 

C *2 • /^1 9 (/^ 2 ) • • • ) Tn) 

are two {n — 1 )-dimensional, analytic, fold bifurcation surfaces of multiplicity- 
two limit cycles of ( 2 . 2 ) which intersect in a cusp along C 3 . 


Definition 3.3. An analytic surface C 4 C is an {n — 3)-dimensional 
swallow-tail bifurcation surface of multiplicity-four limit cycles of ( 2 . 2 ) through 
a point Hq G -R", if for all e > 0 there exists a 5 > 0 such that for each /j, E Ci 
with 11 ^ — /XqII < 6 , system ( 2 . 2 ) has a unique multiplicity-four limit cycle 
in an e-neighborhood of Lq and system ( 2 . 2 ) undergoes a swallow-tail bi¬ 
furcation at L^; i. e., C 4 is the intersection of two (n — 2 )-dimensional cusp 
bifurcation surfaces of multiplicity-three limit cycles which intersect in a 
cusp along C 4 ] furthermore, there are three (n—l)-dimensional fold bifurcation 
surfaces of multiplicity-two limit cycles of ( 2 . 2 ), C 2 *\ f = 0 , 1 , 2 , such that 
and € 2 ^^ intersect in a cusp along Cfi, and intersect in a cusp along 

Cfi, and 6 * 2 ^^ and intersect along an (n—2)-dimensional surface on which 
( 2 . 2 ) has two multiplicity-two limit cycles; finally, for ||/x — ^tg|| < 5 and for /x 
in the swallow-tail region (shaded in Fig. 4), system (2.2) has four simple limit 
cycles in an e-neighborhood of Lq; for ||/x — /Xq|| <5 and pL above the surfaces 
6 * 2 *^ f = 0 , 1 , 2 , system ( 2 . 2 ) has two simple limit cycles in an e-neighborhood 
of Lq; for ||/x —/Xq|| <5 and /x below the surfaces C 2 *\ i = 0 , 1 , 2 , system ( 2 . 2 ) 
has no limit cycles in an e-neighborhood of Lq. Cf. Fig. 4. 


Theorem 3.3. Suppose that n > 4, that for pL = pi^ E RL system (2.2) 
has a multiplicity-four limit cycle Lq, that plq) 7 ^ 0, (0,/Xg) 0, 
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drrfii ( 0 , ^^o) 7 ^ 0 ) for j = 2 ,... ,n, 


d{d, dr) 


(0) Mo) ¥" 0) 


d{d, drr) 


(0) Mo) 7^ O’ 


d{dr, drr) 

^(Mi,Mi) 


(O 5 Mo) 7^ 0- 


Then given e > 0, there is a 6 > t) and constants ujjk = ±1 for j = 2,... ,n, 
fc = 1, 2, and there exist unique functions ( 7 j(/i 2 , • • •, Mn), (M 2 , • • •, Tn) and 
Fi(/i 2 , . . . , Mn), with ^i(M2°\ • • • , Mn ^) = (M2°\ • ■ ■ , Mn ^) = ^*(M2°\ • • • , Mn ^) = 


/i®, for i = 0,1,2 and A; = 1,2, where Fi is defined and analytic for i = 
0,1,2, and \yij — yif^\ < 6, j = 2,... ,n, h^ are defined and continuous for 
0 < ojjkijXj — ix'p) < 6 and analytic for 0 < ojjkiiXj — /x®) < d, j = 2,..., n, 
k = 1,2, and for i = 0, 1,2, gi is defined and analytic in the cuspidal region 
between the surfaces = /if (M 2 , • • • ,Mn), which intersect in a cusp, and Pi is 
continuous in the closure of that region, such that 


Ml = -/^o(M2, • 

• • , t^n) 

Ml = -/^i(M2, • 

■ ■ , Tn) 

Ml = -/^2(M2, • 

■ ■ , Tn) 


is an {n — 3)-dimensional, analytic, swallow-tail bifurcation surface of multi¬ 
plicity-four limit cycles of (2.2) through the point which is the intersection 
of two {n — 2)-dimensional, analytic, cusp bifurcation surfaces of multiplicity- 
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three limit cycles of ( 2 . 2 ), 

I /^1 (/^2) ■ ■ ■ ) /^n) 

[ /il = ■ ■ ■ ,/^n) 

which intersect in a cusp along C 4 ] furthermore, Cf' = fl and Cf = 
<^ 2 °^ n where for i = 0,1, 2, 

C*2 • h'l ■ ■ ■ y P'n) 

are {n — 1 )-dimensional, analytic, fold bifurcation surfaces of multiplicity-two 
limit cycles of ( 2 . 2 ) which intersect in cusps along and in an {n — 2)- 
dimensional, analytic surface on which ( 2 . 2 ) has two multiplicity- 

two limit cycles (Fig. 4 and Fig. 5). 



FIG. 5. The bifurcation curve (one-parameter family) of multiple limit cycles. 


Based on Theorems 2.3, 2.4, the following generalization of Theorems 3.1-3.3 
can be proved on induction [T9] . 


Theorem 3.4. Given m > 2. Suppose that n > m, that for ^ 

polynomial system ( 2 . 2 ) has a multiplicity-m limit cycle Lq, that 


dd 

9/ii 


(O5 A^o) 7^ 0; 


ddr 

dpLi 


(0) Mo) 7^ O’ 


dp. I 


(O’ Mo) 7^ 0, 
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and that 


for i, j = 0,..., m — 2 with i ^ j and A; = 2,..., n. 

Then given £ > 0 there is a 6 > 0 such that for ||^i — ^AqU < 6 , the system (2.2) 
has 

(1) a unique [n — m + 1)-dimensional analytic surface Cm of multiplicity-m 
limit cycles of ( 2 . 2 ) through the point 

(2) two {n — m + 2 )-dimensional analytic surfaces Cm-i of multiplicity-{m—l) 
limit cycles of ( 2 . 2 ) through the point Hq which intersect in a cusp along Cm', 



(j) exactly)., {n—m+j)-dimensional analytic surfaces Cm-j+i of multiplicity- 
(m — j + 1) limit cycles of (2.2) through the point pL^ which intersect pairwise 
in cusps along the bifurcation surfaces Cm-j+ 2 ', 


{m — 1 ) exactly (m — 1 ), {n — 1)-dimensional analytic fold bifurcation surfaces 
C 2 of multiplicity-two limit cycles of (2.2) through the point /Xg which intersect 
pairwise in a cusp along the {n — 2)-dimensional cusp bifurcation surfaces C 3 . 

Let us formulate now the Wintner-Perko termination principle [I9] for poly¬ 
nomial system (2.2). 


Theorem 3.5 (Wintrier—Perko termination principle). Any one-para- 
meter family of multiplicity-m limit cycles of relatively prime polynomial sys¬ 
tem (2.2) can be extended in a unique way to a maximal one-parameter family 
of multiplicity-m limit cycles of (2.2) which is either open or cyclic. 

If it is open, then it terminates either as the parameter or the limit cycles 
become unbounded; or, the family terminates either at a singular point of (2.2), 
which is typically a fine focus of multiplicity m, or on a (compound) separatrix 
cycle of (2.2) which is also typically of multiplicity m. 


The proof of this principle for general polynomial system (2.2) with a vector 
parameter pL G parallels the proof of the planar termination principle for 
the system 

x = P{x,y,\), y = Q{x,y,X) (3.1) 

with a single parameter A G R (see 0 . IH). since there is no loss of generality 
in assuming that system (2.2) is parameterized by a single parameter A; i. e., 
we can assume that there exists an analytic mapping pi{\) of R into R" such 
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that (2.2) can be written as (3.1) and then we can repeat everything, what 
had been done for system (3.1) in [19]. In particular, if A is a held rotation 
parameter of (3.1), the following Perko’s theorem on monotonic families of 
limit cycles is valid; see [T9] . 


Theorem 3.6. If Lq is a nonsingular multiple limit cycle of (3.1) for A = Aq, 
then Lq belongs to a one-parameter family of limit cycles of (3.1); furthermore: 

1 ) if the multiplicity of Lq is odd, then the family either expands or contracts 
monotonically as A increases through Aq; 

2 ) if the multiplicity of Lq is even, then Lq bifurcates into a stable and an 
unstable limit cycle as A varies from Aq in one sense and Lq disappears as A 
varies from Aq in the opposite sense; i. e., there is a fold bifurcation at Aq. 


4 Global bifurcation analysis 


Consider system (1.10). This system has two invariant straight lines: x = 0 
and y = 0. Its hnite singularities are determined by the algebraic system 


x((l — Xx){ax‘^ + /dx + 1) — xy) = 0, 
y{{6 + iJ.y){ax‘^ + /dx + 1) — x^) = 0. 


(4.1) 


From (4.1), we have got: two singular points (0, 0) and (0, —5/p), at most two 
points dehned by the condition 

ax^ + /dx + 1 = 0, 1 / = 0, (4.2) 

and at most six singularities dehned by the system 


xy = {1 — Ax)(q;x^ + /dx + 1), 
y{5 + py) = x{l- Ax), 


(4.3) 


among which we always have the point (1/A, 0). 

The point (0, 0) is always a saddle, but (1/A, 0) can be a node or a saddle, or a 
saddle-node. The point (1/A,0) can change multiplicity when singular points 
enter or exit the hrst quadrant. In addition, a singular point of multiplicity 2 
may appear in the hrst quadrant and bifurcate into two singular points. In the 
case /d > 0 (respectively, /d < 0), there is a possibility of up to one singular 
point (respectively, two singular points) in the open hrst quadrant [T7] . If there 
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exists exactly one simple singular point in the open first quadrant, then it is 
an anti-saddle. If there exists exactly two simple singular points in the open 
hrst quadrant, then the singular point on the left with respect to the a:-axis 
is an anti-saddle and the singular point on the right is a saddle dZ]. If a 
singular point is not in the hrst quadrant, in consequence, it has no biological 
signihcance. 

To study singular points of (1.10) at inhnity, consider the corresponding diffe¬ 
rential equation 


dy y{{6 + yy){ax^ + jdx 1) — x^) 

dx a;((l — \x){ax‘^ + (5x + 1) — xy) 


(4.4) 


Dividing the numerator and denominator of the right-hand side of (4.4) by x^ 
{x 7 ^ 0) and denoting y/xhju (as well as dy/dx), we will get the algebraic 
equation 

X)u — 1) = 0, where u = y/x, (4.5) 

for all inhnite singularities of (4.4) except when x = 0 (the “ends” of the 
y-axis), see n. 0- For this special case we can divide the numerator and 
denominator of the right-hand side of (4.4) by y^ {y ^ 0) denoting x/y hy v 
(as well as dx/dy) and consider the algebraic equation 

n^(n —/i/A) = 0, where v = x/y. (4.6) 

The equations (4.5) and (4.6) give three singular points at inhnity for (4.4): 
a simple node on the “ends” of the a:-axis, a triple node on the “ends” of the 
l/-axis, and a simple saddle in the direction of y/x = A/p. 

To investigate the character and distribution of the singular points in the 
phase plane, we have used a method developed in a-na. The sense of this 
method is to obtain the simplest (well-known) system by vanishing some pa¬ 
rameters (usually held rotation parameters) of the original system and then to 
input these parameters successively one by one studying the dynamics of the 
singular points (both hnite and inhnite) in the phase plane. 

Using the obtained information on singular points and applying a geometric 
approach developed in i-csi, we can study the limit cycle bifurcations of 
system (1.10). This study will use some results obtained in [T7]: in particular, 
the results on the cyclicity of a singular point of (1.10). However, it is surely 
not enough to have only these results to prove the main theorem of this paper 
concerning the maximum number of limit cycles of system (1.10). 

Applying the dehnition of a held rotation parameter 15,0, [To] , i. e., a param¬ 
eter which rotates the held in one direction, to system (1.10), let us calculate 
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the corresponding determinants for the parameters a and /3, respectively: 


Aa = PQ'^ - QP'^ = x^y{y{5 + yy) - x{l - Xx)), (4.7) 

A /3 = PQ'p - QP'p = x'^y{y{5 + yy) - x{l - Ax)). (4.8) 

It follows from (4.7) and (4.8) that on increasing a or /3 the vector field of 
( 1 . 10 ) in the hrst quadrant is rotated in positive direction (counterclockwise) 
only on the outside of the ellipse 

y{5 + yy) — x(l — Ax) = 0. (4.9) 

Therefore, to study limit cycle bifurcations of system (1.10), it makes sense 
together with ( 1 . 10 ) to consider also an auxiliary system ( 1 . 11 ) with a field 
rotation parameter 7 : 

+ q2 > Q_ 

Using system (1.11) and applying Perko’s results, we will prove the following 
theorem. 

Theorem 4.1. System (1.10) can have at most two limit cycles surrounding 
one singular point. 

Proof. First let us prove that system (1.10) can have at least two limit cycles. 
Begin with system (1.10), where a = /? = 0. It is clear that such a cubic 
system, with two invariant straight lines, cannot have limit cycles at all im. 
Inputting a negative parameter (3 into this system, the vector held of (1.10) 
will be rotated in negative direction (clockwise) at inhnity, the structure and 
the character of stability of inhnite singularities will be changed, and an un¬ 
stable limit, Fi, will appear immediately from inhnity in this case. This cycle 
will surround a stable anti-saddle (a node or a focus) A which is in the hrst 
quadrant of system (1.10). Inputting a positive parameter a, the vector held of 
quartic system ( 1 . 10 ) will be rotated in positive direction (counterclockwise) 
at inhnity, the structure and the character of stability of inhnite singulari¬ 
ties will be changed again, and a stable limit, F 2 , surrounding Fi will appear 
immediately from inhnity in this case. On further increasing the parameter 

а, the limit cycles Fi and F 2 combine a semi-stable limit, F 12 , which then 
disappears in a “trajectory concentration” n, 0. Thus, we have proved that 
system ( 1 . 10 ) can have at least two limit cycles; see also [T7] . 

Let us prove now that this system has at most two limit cycles. The proof 
is carried out by contradiction applying Catastrophe Theory; see 0 , na. 
Consider system (1.11) with three parameters: a, /?, and 7 (the parameters 

б , A, and y can be hxed, since they do not generate limit cycles). Suppose 
that (1.11) has three limit cycles surrounding the only point A in the hrst 
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quadrant. Then we get into some domain of the parameters a, jS, and 7 being 
restricted by dehnite conditions on three other parameters S, A, and /x. This 
domain is bounded by two fold bifurcation surfaces forming a cusp bifurcation 
surface of multiplicity-three limit cycles in the space of the parameters a, 13, 
and 7 [5], [T9] . 

The corresponding maximal one-parameter family of multiplicity-three limit 
cycles cannot be cyclic, otherwise there will be at least one point corresponding 
to the limit cycle of multiplicity four (or even higher) in the parameter space. 
Extending the bifurcation curve of multiplicity-four limit cycles through this 
point and parameterizing the corresponding maximal one-parameter family of 
multiplicity-four limit cycles by the held rotation parameter 7 , according to 
Theorem 3.6, we will obtain two monotonic curves of multiplicity-three and 
one, respectively, which, by the Wintner-Perko termination principle (Theo¬ 
rem 3.5), terminate either at the point A or on a separatrix cycle surrounding 
this point. Since we know at least the cyclicity of the singular point which 
is equal to two (see ini). we have got a contradiction with the termination 
principle stating that the multiplicity of limit cycles cannot be higher than 
the multiplicity (cyclicity) of the singular point in which they terminate. 

If the maximal one-parameter family of multiplicity-three limit cycles is not 
cyclic, using the same principle (Theorem 3.5), this again contradicts the 
cyclicity of A (see 113) not admitting the multiplicity of limit cycles to be 
higher than two. This contradiction completes the proof in the case of one 
singular point in the hrst quadrant. 

Suppose that system (1.11) with two hnite singularities, a saddle S and an 
anti-saddle A, has three limit cycles surrounding A. Then we get again into 
some domain of the parameters a, f3, and 7 bounded by two fold bifurcation 
surfaces forming a cusp bifurcation surface of multiplicity-three limit cycles in 
the space of the parameters a, (3, and 7 being restricted by dehnite conditions 
on three other parameters 6, A, and fi 0. Ha- 

The corresponding maximal one-parameter family of multiplicity-three limit 
cycles cannot be cyclic, otherwise there will be at least one point corresponding 
to the limit cycle of multiplicity four (or even higher) in the parameter space. 
Extending the bifurcation curve of multiplicity-four limit cycles through this 
point and parameterizing the corresponding maximal one-parameter family of 
multiplicity-four limit cycles by the held rotation parameter 7 , according to 
Theorem 3.6, we will obtain again two monotonic curves of multiplicity-three 
and one, respectively, which, by Theorem 3.5, terminate either at the point 
A or on a separatrix loop surrounding this point. Since we know at least the 
cyclicity of the singular point which is equal to two (see la). we have got a 
contradiction with the termination principle (Theorem 3.5). 
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If the maximal one-parameter family of multiplicity-three limit cycles is not 
cyclic, using the same principle, this again contradicts the cyclicity of A (see 
HZl) not admitting the multiplicity of limit cycles higher than two. Moreover, 
it also follows from the termination principle that a separatrix loop cannot 
have the multiplicity (cyclicity) higher than two in this case. 

Thus, we conclude that system (1.10) cannot have either a multiplicity-three 
limit cycle or more than two limit cycles surrounding a singular point which 
proves the theorem. □ 
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